migrate xpack-metricbeat #38081

sharbuz · 2024-02-21T10:58:06Z

What is the problem this PR solves?

Jenkins->Buildkite pipelines migration

Migrate x-pack metricbeat pipeline

Example of the pipelines:
https://buildkite.com/elastic/beats-xpack-metricbeat/builds/623

Examples of the affected pipelines:
beats-libbeat
beats-metricbeat
beats-packetbeat
beats-winlogbeat
beats-xpack-libbeat

Related issues

https://github.com/elastic/ingest-dev/issues/1693
https://github.com/elastic/ingest-dev/issues/2993 - related to the mentioned inside the PR issue

botelastic · 2024-02-21T10:58:20Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2024-02-21T11:31:38Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 13 min 20 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

v1v · 2024-03-11T10:02:33Z

.buildkite/scripts/common.sh

+
+cleanup() {
+  echo "---Terraform Cleanup"
+  .ci/scripts/terraform-cleanup.sh "${MODULE_DIR}"              #TODO: move all docker-compose files from the .ci to .buildkite folder before switching to BK


I think, there is a missing context what's the reason for running this terraform clean up function. A comment can help in the future.
In addition, should this particular script run only if the stage cloud ran for the x-pack/metricbeat? Or will it work regardless of the stages in all the beats in case they call the cleanup function?

Both of these functions will be run only when it's needed, for example: https://github.com/sharbuz/beats/blob/25547b4ef8c75e9c356573b39e0bc25c0bf5e022/.buildkite/scripts/cloud_tests.sh#L7
It will run when the script finishes with EXIT code, which means when it finishes successfully/unsuccessfully

I do understand the trap function.
My main concern is regarding the cleanup function itself. If the pipeline finishes with exit 0, then cleanup will run, but if it fails, will the failure be reported, too? I'm not sure if that's what we want. Post-build steps for tearing down resources should not fail the build but make it unstable or notify the cloud resources manager with a message, IMO.

My team enabled a cloud-reaper mechanism that runs async to delete any leftovers in any of the cloud providers we use. That's how we can avoid having stalled resources and failing builds if the cleanup failed.

I've got what you mean, In this case I would suggest using the additional "cleanup" step with Slack notification or pipeline with email notification. @v1v WDYT?

To illustrate a bit more, the current implementation in Jenkins does not fail when tearing down the resources with terraform/docker:

beats/Jenkinsfile

Lines 945 to 950 in 3f46222

dirs?.each { folder ->

// If it failed then cleanup without failing the build

sh(label: 'Terraform Cleanup', script: ".ci/scripts/terraform-cleanup.sh ${folder}", returnStatus: true)

}

// Cleanup the docker services

sh(label: 'Docker Compose Cleanup', script: ".ci/scripts/docker-services-cleanup.sh", returnStatus: true)

It uses returnStatus: true that means do nothing, see

returnStatus : boolean (optional)
Normally, a script which exits with a nonzero status code will cause the step to fail with an exception. If this option is checked, the return value of the step will instead be the status code. You may then compare it to zero, for example.

See https://www.jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script

You can get the same:
either you can || true or use your additional cleanup with soft_fail in Buildkite.

WDYT?

Sounds good to me. But let me ask @dliappis about his preference.

done: f432822

.buildkite/scripts/common.sh

v1v

There are a few open discussions without an explicit answer or action, hence I'm blocking this.

.buildkite/scripts/common.sh

sharbuz · 2024-03-11T07:13:55Z

.buildkite/scripts/install_tools.sh

@@ -43,6 +43,10 @@ with_dependencies
 config_git
 mage dumpVariables

+if [[ "$BUILDKITE_PIPELINE_SLUG" == "beats-xpack-metricbeat" && "${BUILDKITE_STEP_KEY}" == "extended-cloud-test" ]]; then
+  startCloudTestEnv "${MODULE_DIR}"


The reason for using this approach is to collect all needed dependencies/tools. When it's done we'll think about using our own BK images with one/two/three commands in the command step. For example:

beats/.buildkite/scripts/generate_xpack_metricbeat_pipeline.sh

Line 170 in 48e1c08

command: "cd $BEATS_PROJECT_NAME && mage package"

sharbuz · 2024-03-11T07:14:07Z

.buildkite/scripts/common.sh

+if [[ "$BUILDKITE_PIPELINE_SLUG" == "beats-xpack-metricbeat" ]]; then
+  withModule "${MODULE_DIR}"
 fi


answered here

sharbuz · 2024-03-11T07:14:27Z

.buildkite/scripts/setenv.sh

  exportVars
  export RACE_DETECTOR="true"
  export TEST_COVERAGE="true"
  export DOCKER_PULL="0"
+  export TEST_TAGS="oracle"


fixed/changed

.buildkite/scripts/setenv.sh

.buildkite/hooks/pre-command

sharbuz · 2024-03-11T10:08:13Z

.buildkite/scripts/install_tools.sh

@@ -43,6 +43,10 @@ with_dependencies
 config_git
 mage dumpVariables

+if [[ "$BUILDKITE_PIPELINE_SLUG" == "beats-xpack-metricbeat" && "${BUILDKITE_STEP_KEY}" == "extended-cloud-test" ]]; then
+  startCloudTestEnv "${MODULE_DIR}"


I have tested your suggestions and I faced some issues and I decided to revert my changes. I will back to this on Phase 2.

If it's placed in the cloud_tests.sh it doesn't work because the metricbeat-pipeline step requires the defined MODULE variable.

If it's placed in the pre-command hook it requires huge structure changes or duplicating the code.
In my opinion, we don't have enough time for this now and we have to focus on the main functionality. When pipeline generator is ready - we'll think about optimization and logic again.
@v1v @dliappis WDYT?

sharbuz · 2024-03-11T10:13:25Z

.buildkite/scripts/common.sh

+
+cleanup() {
+  echo "---Terraform Cleanup"
+  .ci/scripts/terraform-cleanup.sh "${MODULE_DIR}"              #TODO: move all docker-compose files from the .ci to .buildkite folder before switching to BK


Both of these functions will be run only when it's needed, for example: https://github.com/sharbuz/beats/blob/25547b4ef8c75e9c356573b39e0bc25c0bf5e022/.buildkite/scripts/cloud_tests.sh#L7
It will run when the script finishes with EXIT code, which means when it finishes successfully/unsuccessfully

.buildkite/scripts/common.sh

.buildkite/hooks/pre-command

.buildkite/scripts/install_tools.sh

v1v · 2024-03-12T10:26:09Z

.buildkite/scripts/cloud_tests.sh

+trap 'teardown; unset_secrets' EXIT
+
+# Set the MODULE env variable if possible
+defineModuleFromTheChangeSet "${MODULE_DIR}"


IIUC, this particular function should use the beats project folder.

directory in Jenkinsfile is set in

beats/Jenkinsfile

Line 1133 in 3f46222

directory: args.project,

and

beats/Jenkinsfile

Line 1105 in 3f46222

* - project -> the name of the project that should match with the folder name.

explains what the project means.

Then directory is used in

beats/Jenkinsfile

Line 676 in 3f46222

def module = withModule ? getCommonModuleInTheChangeSet(directory) : ''

I've changed the function and its call: 141c990 and 3cf8aa0

v1v · 2024-03-12T10:31:31Z

.buildkite/hooks/pre-command

+  export BEATS_AWS_SECRET_KEY
+  BEATS_AWS_ACCESS_KEY=$(retry 5 vault kv get -field access_key ${AWS_SERVICE_ACCOUNT_SECRET_PATH})
+  export BEATS_AWS_ACCESS_KEY
+fi


After thinking a bit more about defineModuleFromTheChangeSet and how other places could use it. I think MODULE could be set in the pre-command hook.

Suggested change

fi

# Set the MODULE env variable

defineModuleFromTheChangeSet "${BEATS_PROJECT_NAME}"

Therefore my former suggestion for adding defineModuleFromTheChangeSet in .buildkite/scripts/cloud_tests.sh could be deleted.

I tested it here: c96b847
And if we move that right now to the pre-command, it will require moving to pre-command all dependencies, answered here

Then, with this approach, the MODULE env variable won't be set in the other stages. Hence, it will take longer to run the tests.

If that's something to be done in phase2 that's totally fine with me. But PRs will be slow to be tested. Unless the defineModuleFromTheChangeSet "${BEATS_PROJECT_NAME}" is also called in all those tests using the withModule: true

elasticmachine · 2024-03-12T13:40:09Z

💔 Build Failed

Buildkite Build
Commit: f432822

Failed CI Steps

History

💔 Build #3633 failed 3cf8aa0
💔 Build #3623 failed 84e8305
💔 Build #3578 failed c96b847
💔 Build #3577 failed 25547b4

cc @sharbuz

elasticmachine · 2024-03-12T13:40:19Z

💔 Build Failed

Buildkite Build
Commit: f432822

Failed CI Steps

History

💔 Build #2292 failed 3cf8aa0
💔 Build #2282 failed 84e8305
💔 Build #2237 failed c96b847
💔 Build #2236 failed 25547b4

cc @sharbuz

elasticmachine · 2024-03-12T13:57:45Z

💚 Build Succeeded

Buildkite Build
Commit: f432822

History

💚 Build #1985 succeeded 84e8305
💚 Build #1940 succeeded c96b847
💔 Build #1939 failed 25547b4
💚 Build #1935 succeeded 1ca09df
💚 Build #1930 succeeded fd09867
💚 Build #1885 succeeded 48e1c08

cc @sharbuz

v1v

As I mentioned, IIUC, the current MODULE env variable won't be set hence ITs will take longer on PRs.

If that's something to be revisited in the future, then LGTM. But if build times are important for phase 1... I'd suggest to do a follow-up to set MODULE env variable in the required stages.

v1v · 2024-03-12T14:01:42Z

.buildkite/hooks/pre-command

+  export BEATS_AWS_SECRET_KEY
+  BEATS_AWS_ACCESS_KEY=$(retry 5 vault kv get -field access_key ${AWS_SERVICE_ACCOUNT_SECRET_PATH})
+  export BEATS_AWS_ACCESS_KEY
+fi


Then, with this approach, the MODULE env variable won't be set in the other stages. Hence, it will take longer to run the tests.

If that's something to be done in phase2 that's totally fine with me. But PRs will be slow to be tested. Unless the defineModuleFromTheChangeSet "${BEATS_PROJECT_NAME}" is also called in all those tests using the withModule: true

elasticmachine · 2024-03-12T14:07:14Z

💚 Build Succeeded

Buildkite Build
Commit: f432822

History

💚 Build #1148 succeeded 84e8305
💚 Build #1103 succeeded c96b847
💔 Build #1102 failed 25547b4
💚 Build #1098 succeeded 1ca09df
💚 Build #1093 succeeded fd09867
💚 Build #1048 succeeded 48e1c08

cc @sharbuz

elasticmachine · 2024-03-12T14:08:06Z

💚 Build Succeeded

Buildkite Build
Commit: f432822

History

💚 Build #1992 succeeded 84e8305
💔 Build #1947 failed c96b847
💔 Build #1946 failed 25547b4
💚 Build #1942 succeeded 1ca09df
💚 Build #1937 succeeded fd09867
💛 Build #1892 was flaky 48e1c08

cc @sharbuz

elasticmachine · 2024-03-12T14:26:02Z

💚 Build Succeeded

Buildkite Build
Commit: f432822

History

💚 Build #3199 succeeded 84e8305
💚 Build #3154 succeeded c96b847
💔 Build #3153 failed 25547b4
💚 Build #3149 succeeded 1ca09df
💚 Build #3099 succeeded 48e1c08
💚 Build #3085 succeeded 9907579

cc @sharbuz

* migrate xpack-metricbeat (cherry picked from commit 843010c)

* migrate xpack-metricbeat (cherry picked from commit 843010c) # Conflicts: # .buildkite/scripts/generate_packetbeat_pipeline.sh

* migrate xpack-metricbeat (cherry picked from commit 843010c)

* migrate xpack-metricbeat (cherry picked from commit 843010c) Co-authored-by: sharbuz <[email protected]>

* migrate xpack-metricbeat (#38081) * migrate xpack-metricbeat (cherry picked from commit 843010c) # Conflicts: # .buildkite/scripts/generate_packetbeat_pipeline.sh * Update generate_packetbeat_pipeline.sh * Update setenv.sh --------- Co-authored-by: sharbuz <[email protected]>

migrate xpack-metricbeat

2b5dd7b

sharbuz self-assigned this Feb 21, 2024

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 21, 2024

change pre-command

cf44101

Siarhei Harbuz added 15 commits February 21, 2024 14:41

add mage_version

e432525

add mage_version

0783eee

disable some trigger steps

2a1de26

change command with mage

7e55863

add ASDF_MAGE_VERSION

cf55e5b

fix arm image name

def5673

fix macos

85bfec8

test arm packaging

5af9510

test aws/macos

90bec63

test macos

bd038c2

test macos

77b29b2

test macos

5b41d3b

test macos

2751e38

test macos

88800c8

fix macos declare

4805ccb

revert all last changes

c96b847

v1v reviewed Mar 11, 2024

View reviewed changes

sharbuz requested a review from v1v March 11, 2024 10:32

v1v requested changes Mar 11, 2024

View reviewed changes

sharbuz commented Mar 12, 2024

View reviewed changes

v1v reviewed Mar 12, 2024

View reviewed changes

.buildkite/scripts/install_tools.sh Outdated Show resolved Hide resolved

Siarhei Harbuz added 2 commits March 12, 2024 12:00

test Victor's suggestion

1057a94

test Victor's suggestion

84e8305

v1v reviewed Mar 12, 2024

View reviewed changes

Siarhei Harbuz added 3 commits March 12, 2024 14:18

change defineModuleFromTheChangeSet

141c990

change defineModuleFromTheChangeSet

3cf8aa0

add true for teardown

f432822

v1v approved these changes Mar 12, 2024

View reviewed changes

sharbuz merged commit 843010c into elastic:main Mar 12, 2024
29 of 33 checks passed

mergify bot pushed a commit that referenced this pull request Mar 12, 2024

migrate xpack-metricbeat (#38081)

286b2ab

* migrate xpack-metricbeat (cherry picked from commit 843010c)

mergify bot pushed a commit that referenced this pull request Mar 12, 2024

migrate xpack-metricbeat (#38081)

dd8b27e

* migrate xpack-metricbeat (cherry picked from commit 843010c) # Conflicts: # .buildkite/scripts/generate_packetbeat_pipeline.sh

This was referenced Mar 12, 2024

[7.17](backport #38081) migrate xpack-metricbeat #38271

Merged

[8.12](backport #38081) migrate xpack-metricbeat #38272

Merged

mergify bot pushed a commit that referenced this pull request Mar 12, 2024

migrate xpack-metricbeat (#38081)

64ba153

* migrate xpack-metricbeat (cherry picked from commit 843010c)

mergify bot mentioned this pull request Mar 12, 2024

[8.13](backport #38081) migrate xpack-metricbeat #38273

Merged

sharbuz added a commit that referenced this pull request Mar 12, 2024

migrate xpack-metricbeat (#38081) (#38273)

853e4a2

* migrate xpack-metricbeat (cherry picked from commit 843010c) Co-authored-by: sharbuz <[email protected]>

sharbuz added a commit that referenced this pull request Mar 12, 2024

migrate xpack-metricbeat (#38081) (#38272)

a0c1646

* migrate xpack-metricbeat (cherry picked from commit 843010c) Co-authored-by: sharbuz <[email protected]>

sharbuz deleted the migrate-xpack-metricbeat branch March 12, 2024 15:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate xpack-metricbeat #38081

migrate xpack-metricbeat #38081

sharbuz commented Feb 21, 2024 •

edited

Loading

botelastic bot commented Feb 21, 2024

elasticmachine commented Feb 21, 2024 •

edited

Loading

Build stats

v1v Mar 11, 2024

sharbuz Mar 11, 2024

v1v Mar 12, 2024

sharbuz Mar 12, 2024 •

edited

Loading

v1v Mar 12, 2024

sharbuz Mar 12, 2024

v1v left a comment

sharbuz Mar 11, 2024

sharbuz Mar 11, 2024

sharbuz Mar 11, 2024

sharbuz Mar 11, 2024

sharbuz Mar 11, 2024

v1v Mar 12, 2024

sharbuz Mar 12, 2024 •

edited

Loading

v1v Mar 12, 2024

sharbuz Mar 12, 2024

v1v Mar 12, 2024

elasticmachine commented Mar 12, 2024 •

edited

Loading

elasticmachine commented Mar 12, 2024 •

edited

Loading

elasticmachine commented Mar 12, 2024

v1v left a comment

v1v Mar 12, 2024

elasticmachine commented Mar 12, 2024

elasticmachine commented Mar 12, 2024

elasticmachine commented Mar 12, 2024

	dirs?.each { folder ->
	// If it failed then cleanup without failing the build
	sh(label: 'Terraform Cleanup', script: ".ci/scripts/terraform-cleanup.sh ${folder}", returnStatus: true)
	}
	// Cleanup the docker services
	sh(label: 'Docker Compose Cleanup', script: ".ci/scripts/docker-services-cleanup.sh", returnStatus: true)

migrate xpack-metricbeat #38081

migrate xpack-metricbeat #38081

Conversation

sharbuz commented Feb 21, 2024 • edited Loading

What is the problem this PR solves?

Related issues

botelastic bot commented Feb 21, 2024

elasticmachine commented Feb 21, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sharbuz Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

v1v left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sharbuz Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Mar 12, 2024 • edited Loading

💔 Build Failed

Failed CI Steps

History

elasticmachine commented Mar 12, 2024 • edited Loading

💔 Build Failed

Failed CI Steps

History

elasticmachine commented Mar 12, 2024

💚 Build Succeeded

History

v1v left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Mar 12, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 12, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 12, 2024

💚 Build Succeeded

History

sharbuz commented Feb 21, 2024 •

edited

Loading

elasticmachine commented Feb 21, 2024 •

edited

Loading

sharbuz Mar 12, 2024 •

edited

Loading

sharbuz Mar 12, 2024 •

edited

Loading

elasticmachine commented Mar 12, 2024 •

edited

Loading

elasticmachine commented Mar 12, 2024 •

edited

Loading